Motion robust depth estimation using convolution and wavelet transforms

ABSTRACT

Apparatus and method for electronically estimating focusing distance between a camera (still and/or video camera) and a subject. Images at different focal positions of a calibration target are collected to arrive at a focus matching model for a given imaging apparatus. In operation, at least two images are captured and convolutions performed which approximate the modeling of blur change as a point spread function. Wavelet transforms are applied to the images after each convolution and images are compared based on the wavelet variance differences to provide a motion robust blur difference determination. Applying the blur differences to the focus matching model provides an estimate of focusing distance, which can be utilized such as for controlling camera focus.

CROSS-REFERENCE TO RELATED APPLICATIONS

Not Applicable

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable

NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION

A portion of the material in this patent document is subject tocopyright protection under the copyright laws of the United States andof other countries. The owner of the copyright rights has no objectionto the facsimile reproduction by anyone of the patent document or thepatent disclosure, as it appears in the United States Patent andTrademark Office publicly available file or records, but otherwisereserves all copyright rights whatsoever. The copyright owner does nothereby waive any of its rights to have this patent document maintainedin secrecy, including without limitation its rights pursuant to 37C.F.R. §114.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention pertains generally to image acquisition and processing,and more particularly to depth estimation.

2. Description of Related Art

Proper camera focus is a critical metric when capturing an image with animage acquisition device (video or still image). Numerous systems havebeen developed for estimating or attaining a proper camera focus. As acamera-lens system has a number of related elements and characteristics,a brief discussion follows of these elements and their associatedcharacteristics.

Generally speaking, the two main optical parameters of a photographiclens are maximum aperture and focal length. The focal length determinesthe angle of view, and the size of the image relative to that of theobject (subject) for a given distance to the subject (subject-distance).The maximum aperture (f-number, or f-stop) limits the brightness of theimage and the fastest shutter speed usable for a given setting (focallength/effective aperture), with a smaller number indicating that morelight is provided to the focal plane which typically can be thought ofas the face of the image sensor in a simple digital camera.

One form of typical simple lens (technically a lens having a singleelement) is that of having a single focal length (also referred to as a“prime lens”). In focusing a camera using a single focal length lens,the distance between lens and the focal plane is changed thereinaltering the focal point of the photographic subject onto that focalplane. So although the single focal length lens has a fixed opticalrelation and focal length, it is used in the camera to focus on subjectsacross a focal range span. Consequently, one should not confuse thefixed focal distance of a lens with the range of focal distanceobtainable on a camera using that lens, whereby adjusting the positionof that lens in relation to the focal plane alters focal distance.

In using a single focal length lens one would adjust aperture to selectthe amount of light with respect to desired shutter speed, and thenadjust focus according to the subject-distance, which is also referredto as the focal distance and then capture an image. Often a macrosetting is provided with a different focal length selection, on anotherwise single focal length lens, for taking close-up shots. Atelephoto lens provides a very narrow angle of view with highmagnification for filling the frame with images from distance objects.

It will be noted that multi-focal length lenses are usually referred toas “zoom” lenses, because image magnification can be “zoomed”, or“unzoomed” as the case may be. Zoom lenses allow the user to select theamount of magnification of the subject, or put another way, the degreeto which the subject fills the frame. It is important to understand thatthe zoom function of these lenses, or camera-lens systems, isconceptually separate from both the focus control and the aperturecontrol.

Irrespective of whether a single-focal length lens or multi-focal lengthlens is utilized, it is necessary to properly focus the lens for a givensubject-distance. An acceptable range of focus for a given focus settingis referred to as “depth of field” which is a measurement of depth ofacceptable sharpness in the object space, or subject space. For example,with a subject distance of fifteen feet, an acceptable range of focusfor a high definition camera may be on the order of inches, whileoptimum focus can require even more precision. It will be appreciatedthat depth of field increases as the focusing moves from intermediatedistances out toward “infinity” (e.g., capturing images of distantmountains, clouds and so forth), which of course at that range hasunlimited depth of field.

For a single focal length lens at a given aperture setting there will bea single optimum focus setting for a given distance from camera to thesubject (subject-distance). Portions of the subject which are closer orfarther than the focal distance of the camera will show up in thecaptured images subject to some measure of blurring, as depends on manyfactors that impact depth of field. However, in a multi-focal lens thereis an optimum focus point for each lens magnification (lens focallength) obtainable by the lens. To increase practicality, lens makershave significantly reduced the need to refocus in response to zoomsettings, however, the necessity for refocusing depends on the specificcamera-lens system in use. In addition, the aperture setting can requirechanging in response to different levels of zoom magnification.

In early camera systems, focus could only be determined and corrected inresponse to operator recognition and a manual focus adjustment. However,due to the critical nature of focus on results, focusing aids werereadily adopted. More recently, imaging devices often provide theability to automatically focus on the subject, a function which isgenerically referred to today as “auto focus”. Focus continues to be apoint of intense technical development as each of the many existing autofocus mechanisms are subject to shortcomings and tradeoffs.

Although numerous focusing mechanisms exist, these can be divided intotwo general types of auto focus (AF) systems: (1) active auto focus and(2) passive auto focus. In active auto focus, one or more image sensorsis utilized to determine distance to the focal point, or otherwise todetect focus external of the image capture lens system. Active AFsystems can perform rapid focusing although they will not typicallyfocus through windows, or in other specific applications, since soundwaves and infrared light are reflected by the glass and other surfaces.In passive auto focus systems the characteristics of the viewed imageare used to detect and set focus.

The majority of high-end SLR cameras currently use through-the-lensoptical AF sensors, which for example, may also be utilized as lightmeters. The focusing ability of these modern AF systems can often be ofhigher precision than that achieved manually through an ordinaryviewfinder.

In one form of passive AF system, phase detection is utilized, such asby dividing the incoming light through a beam splitter into pairs ofimages and comparing them on an AF sensor. Two optical prisms capturethe light rays coming from the opposite sides of the lens and divert itto the AF sensor, creating a simple rangefinder with a base identical tothe diameter of the lens. Focus is determined in response to checkingfor similar light intensity patterns and phase difference calculated todetermine if the object is considered in front of the focus or in backof the proper focus position.

In another type of passive AF system, contrast measurements are madewithin a sensor field through the lens. The system adjusts focus tomaximize intensity difference between adjacent pixels which is generallyindicative of correct image focus. Thus, focusing is performed until amaximum level of contrast is obtained. This form of focusing is slowerthan active AF, in particular when operating under dim light, but is acommon method utilized in low end imaging devices.

Passive systems are notoriously poor at making focal decisions in lowcontrast conditions, notably on large single-colored surfaces (solidsurface, sky, and so forth) or in low-light conditions. Passive systemsare dependent on a certain degree of illumination to the subject(whether natural or otherwise), while active systems may focus correctlyeven in total darkness when necessary.

Accordingly, a need exists for improved auto focusing techniques whichprovide rapid and accurate subject-distance estimations and/or focuscontrol under a wide range of conditions. The present invention fulfillsthat need as well as others and overcomes shortcomings of previouscamera focus techniques.

BRIEF SUMMARY OF THE INVENTION

A method of camera depth estimation is described which is based on blurdifferences and multiple picture matching. This method computes a blurdifference between images captured at different focus positions. Thepresent invention utilizes a novel mechanism for determining thedifference (blur) between images. Previous systems utilized some form ofnorm operator applied to the pixel-to-pixel difference between thoseimages to compare the two images after convolution. In the inventivetechnique, a wavelet transform is applied to the images afterconvolution, and is followed by a calculation of variance of the waveletcoefficients. A difference of the two variances is then determined, suchas preferably an absolute difference. As a consequence not needing tocompare each pixel in a first image to a corresponding pixel position ina second image, apparatus and methods according to the invention providea motion robust form of blur determination which is described hereinwithin a system for estimating subject depth.

It should be appreciated that blur difference varies depending on lensfocus and position in relation to the target image, which according tothe present invention can be approximated using a polynomial model, suchas preferably of at least two-dimensions. The polynomial model iscalibrated, preferably off-line, such as by using a series of step-edgeimages, or similarly convenient calibration image mechanism forregistering proper focus, and is then utilized for calculating the depthfor images for a given image collection apparatus (e.g., camera make ormodel). Accordingly, the calibration target or subject is utilized in acharacterization process in which the blur characteristics (focuscharacteristics) of the camera and lens system are determined andmodeled for use during camera operations.

The discussion herein is directed primarily to a camera having a singlefocal length lens, however, the technique is applicable to multi-focallength lenses (e.g., “zoom” lenses) as will be discussed near the end ofthe specification. It will be appreciated that in addition to autofocusing, the depth estimation taught herein has many applications inareas including computer/robotic vision, surveillance, 3D imaging, andsimilar imaging systems.

According to a general description of the invention, matching curves areobtained for calibration images (e.g., step-edge images) at differentdistances across the whole focusing range, or a desired portion thereof.Then a multi-dimensional (e.g., two-dimensional) model is created,preferably a polynomial model, to represent the matching curves. Thetwo-dimensional polynomial model can then be used for depth estimationwhen blur differences are computed on general images for the givenapparatus.

The following terms are generally described in relation to thespecification, and are not to be interpreted toward constrainingspecific recitations of the specification.

The term “histogram” is a statistical term describing a graphicaldisplay of tabulated frequencies, and generally shows proportionally thenumber of cases falling into each of several categories, whetherdiscrete in bars or across a range.

The term “polynomial” as applied for modeling a matching curve is apolynomial function, such as having the general one dimensional form:

y=a _(n) x ^(n) +a _(n-1) x ^(n-1) + . . . +a ₂ x ² +a ₁ x ¹ +a ₀

in which n is a non-negative integer that defines the degree of thepolynomial. It will be noted that a polynomial with a degree of 3 is acubic, 2 is a quadratic, 1 is a line and 0 is a constant. Polynomialequations can be used for modeling a wide range of empiricallydetermined relationships.

The term “convolution” as used herein describes a mathematical operationon two functions to produce a third function that is typically viewed asa modified version of one of the original functions. Often the secondfunction is reversed and overlays a portion of the first function,toward more properly modeling a given data set.

The term “point spread function” (PSF) describes the response of animaging system to a point source or point object, this is often alsoreferred to as an impulse response, such as found across a step edge. Inthis context, the degree of spreading (blurring) of the point object isa measure for the focal quality of the imaging system.

The term “outlier” is a statistical term indicating that one or moreobservations in the empirical data set are numerically distinct orseparate from the remainder of the data set. Outlier points may indicatesystemic shortcomings, faulty data, and so forth, although a smallnumber of outliers are expected in any large sample sets. Attempting tomodel the data set including the “outliers” could lead to a misleadingmodel, wherein they are typically discarded once it is assured they donot properly represent the characteristics of the underlying function.

The invention is amenable to being embodied in a number of ways,including but not limited to the following descriptions.

One embodiment of the invention is an image capture apparatus,comprising: (a) an imaging device (or image source) as a means forobtaining an image; (b) a computer processor coupled to the imagingdevice (or source); (c) memory coupled to the computer processorconfigured for retaining programming executable on the computerprocessor, wherein said computer and memory are a means for processingimages; (d) a focus matching model retained in the memory, and (e)programming executable on the computer processor for carrying out thesteps of, (e)(i) capturing (or receiving) multiple object images, whichis a means for obtaining object images, (e)(ii) performing convolutionsto model blur changes as a point spread function between the multipleobject images, (e)(iii) determining blur difference within eachconvolution in response to performing a wavelet transform, obtainingwavelet variance and comparing differences (e.g., absolute differences)of wavelet variance for the multiple object images, and (e)(iv)performing depth estimation in response to the convolutions within thefocus matching model.

At least one embodiment of the invention utilizes a focus matching modelbased on imaging calibration targets obtained at different focallengths. At least one embodiment of the invention is configured toutilize at least one size kernel when performing convolutions. Forexample, larger kernels may be used first to speed convergence, withprogressively smaller kernels utilized to obtain the desired level ofaccuracy. At least one embodiment of the invention is configured fordetermining differences of wavelet variance, preferably an absolutedifference of wavelet variances. At least one embodiment of theinvention is configured for determining wavelet variance in at least onewavelet subband and at least one wavelet transform level. At least oneembodiment of the invention is configured for determining waveletvariance in all wavelet subbands in at least one wavelet transformlevel. At least one embodiment of the invention is configured with thefocus matching model utilizing a polynomial function, of any desireddegree, to reduce mismatching noise. At least one embodiment of theinvention is configured with coefficients of the polynomial functionstored in memory.

At least one embodiment of the invention is configured for performingoptional histogram matching of the object images to reduce noise fromoutliers between focal positions prior to inputting the blur differencesinto the focus matching model. Histogram matching need not be used inapplications subject to significantly complex motions. At least oneembodiment of the invention is configured for performing histogrammatching in response to the steps comprising: (a) sequentially shiftingpixels from a first histogram to a second histogram to equalize thepixels of closest luminance; and (b) approximating histogram matchingutilizing a linear matching function; wherein noise effects are reducedwhich have been introduced into the focus matching model in response toundesired physical and environmental variations.

At least one embodiment of the invention is has a focus matching modelgenerated in response to performing a calibration process on theapparatus in which a series of calibration target images are obtainedfor registering proper focus, with focus curves being obtained for theseries of calibration target images; and a multi-dimensional modelgenerated based on matching the focus curves for the series ofcalibration target images. At least one embodiment of the invention isconfigured with the imaging device comprising a still image camera, avideo image camera, or a combination still and video image camera. Atleast one embodiment of the invention further comprises: (a) a focuscontrol element coupled to the imaging device; (b) programmingexecutable on the computer processor for adjusting the focus controlelement in response to performing depth estimation on object imagesbased on inputting blur differences detected between object images intothe focus matching model.

One embodiment of the invention is an image capture apparatus,comprising: (a) an imaging device; (b) a computer processor coupled tothe imaging device; (c) memory coupled to the computer processorconfigured for retaining programming executable on the computerprocessor; (d) a focus matching model based on imaging calibrationtargets at different focal lengths which is retained in the memory, and(e) programming executable on the computer processor for carrying outthe steps of, (e)(i) capturing multiple object images, (e)(ii)performing convolutions by at least one size of convolution kernel tomodel blur changes as a point spread function between the multipleobject images, (e)(iii) determining blur difference within eachconvolution in response to performing wavelet transform, obtainingwavelet variance and comparing differences of wavelet variance in atleast one wavelet subband and at least one wavelet transform level, forthe multiple object images, and (e)(iv) performing depth estimation inresponse to the convolutions within the focus matching model.

One embodiment of the invention is a method of automatic estimation ofcamera-to-object focal depth, comprising: (a) generating amulti-dimensional focus matching model in response to detecting blurdifferences between multiple images of a calibration subject captured atdifferent focal distances; (b) capturing multiple object images; (c)determining blur differences between the multiple object images inresponse to convolutions which model blur changes as a point spreadfunction between the multiple object images; (d) determining blurdifference within each convolution in response to performing wavelettransform, obtaining wavelet variance and comparing differences ofwavelet variance for the multiple object images, and (e) performingdepth estimation in response to the convolutions within the focusmatching model.

One embodiment of the invention is an apparatus and method fordetermining blur difference between images in response to: (a)performing convolutions by at least one size of convolution kernel tomodel blur changes as a point spread function between the multipleobject images; and (b) determining blur difference within eachconvolution in response to performing wavelet transform, obtainingwavelet variance, and comparing differences of wavelet variance in atleast one wavelet subband and at least one wavelet transform level, forthe multiple object images.

The present invention provides a number of beneficial elements which canbe implemented either separately or in any desired combination withoutdeparting from the present teachings.

An element of the invention is a method for estimating distance to asubject (subject-distance, or camera focal distance) in response tocapturing multiple images (or otherwise obtaining images) and matchingmultiple images during characterization of the camera-lens system atmultiple focal points.

Another element of the invention is the use of distance estimation toestimate focus, or to control focus adjustments, within a camera system.

Another element of the invention is a subject-distance estimation methodwhich can estimate distance in response to the input of imagesrepresenting at least two focal settings taken of an image at a givensubject-distance.

Another element of the invention is the use of an image comparisonprocess which is motion robust, as it does not perform a comparison ofpixels in corresponding positions in the images being compared.

Another element of the invention is a subject-distance estimation methodwhich determines blur difference based on wavelet transforms applied tothe subject images whose variance is compared.

Another element of the invention is a subject-distance estimation methodwhich only requires the use of two image inputs for estimating asubject-distance, although additional inputs can be utilized forincreasing estimation accuracy as desired, or for successive and/orcontinuous estimations.

Another element of the invention is a subject-distance estimation methodin which multiple images with different focal settings are captured ofan image having a fixed subject-distance, and blur information fromthese images is plugged into (processed through or by) the focusmatching model which is solved for distance to generate an estimate ofthe actual subject-distance.

Another element of the invention is a subject-distance estimating methodor apparatus which adopts a polynomial model to represent the empiricalfocus matching model.

Another element of the invention is a histogram matching method in whichpixels are sequentially shifted from one histogram to the other toequalize the pixels of closest luminance in the other histogram toreduce the effects of noise introduced into the model in response toundesired physical and environmental variations, and is approximated bya linear matching function.

Another element of the invention is a subject-distance estimationapparatus and method which can be utilized for single focal pointlenses, discrete focal point lenses (e.g., normal and macro settings),or continuously variable focal point lenses (e.g., zoom lenses).

Another element of the invention is a distance estimation apparatus andmethod in which a focus matching model can be generated for eachdiscrete magnification setting of a camera, or at incremental positionsalong a continuously variable magnification (zoom) range.

A still further element of the invention is that depth estimation andfocus can be determined for a wide range of imaging apparatus (e.g.,still and/or video camera devices) configured for capturing images atdifferent focus and zoom settings.

Further elements of the invention will be brought out in the followingportions of the specification, wherein the detailed description is forthe purpose of fully disclosing preferred embodiments of the inventionwithout placing limitations thereon.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The invention will be more fully understood by reference to thefollowing drawings which are for illustrative purposes only:

FIG. 1 is a schematic of capturing multiple images at multiple focalpoints according to an element of the present invention.

FIG. 2A and FIG. 2B are schematic comparisons of calibration target(e.g., step edge) images according to an element of the presentinvention.

FIG. 3 is a schematic of computing blur difference in three iterationsaccording to an element of the present invention.

FIG. 4 is a graph of a matching curve collected according to an elementof the present invention and showing the inclusion of outliers andnoise.

FIG. 5 is a pyramid representation of a wavelet transform utilizedaccording to an element of the present invention, showing subbands and athree level transform structure.

FIG. 6 is a flowchart of wavelet-based blur difference determinationaccording to an embodiment of the present invention.

FIG. 7 is a histogram of mismatching between successivesubject-distances according to an element of the present invention.

FIG. 8 is a magnified histogram showing a portion of the histogramdepicted in FIG. 7.

FIG. 9 is a graph of a matching curve showing matching before and afterhistogram matching according to the present invention.

FIG. 10 is a graph of a matching curve across fifteen differentdistances showing matching before and after histogram matching accordingto an element of the present invention.

FIG. 11 is a graph of a matching curve showing the use of bi-quadraticfitting according to an element of the present invention.

FIG. 12 is a graph of a matching curve showing the use of bi-cubicfitting according to an element of the present invention.

FIG. 13 is a flowchart of calibration according to an element of thepresent invention.

FIG. 14 is a flowchart of camera depth estimation based on two picturematching according to an element of the present invention.

FIG. 15 is a flowchart of histogram matching according to an element ofthe present invention.

FIG. 16 is a flowchart of depth estimation according to an embodiment ofthe present invention.

FIG. 17 is a block diagram of an image capture apparatus configured forperforming depth estimation according to an element of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

1. Blur Difference.

In considering blur differences between images, it will be recognizedthat when a subject is in optimal focus, the captured image is thesharpest and accordingly provides the highest contrast in relation toimages captured at less than optimal focus. The subject becomesincreasingly blurry (less contrast) as the lens moves away from thein-focus position. Generally, when two pictures are captured (taken) ofa specific subject at two different focus distances, the image capturedcloser to the subject distance is sharper than the other. The focusdistances at which the pictures are taken and the amount of blurdifference between these two pictures can be used in a proper model,calibrated for the specific camera model and/or make, to estimate theactual subject distance, or depth. In the present invention this depthestimation is performed in response to a polynomial model.

FIG. 1 illustrates an embodiment 10 in which multiple images arecaptured of a calibration target (or calibration subject), at differentfocal positions (subject-distances) when collecting a data set for agiven imaging apparatus (e.g., specific embodiment, make or model ofcamera, or a family of cameras using the same/similar optical imagingelements). Collecting the data set comprises a characterization processfor the camera-lens system at a given magnification setting (lens at afixed focal length, zoom setting). An imaging device (camera) 12 isshown which can focus from a minimum focal length 14 on out to infinity16. Minimum focal distance 14 (e.g., in this case 35 cm) is shown aswell as focus at infinity 16. According to the invention, the focusconverges to first focal position 18 and then to a second focal position20, upon a calibration target 22, such as step-edge image, slate,graticule, or similar target having known optical characteristics, alongfocal path 24.

By way of example and not limitation, a Sony DSC-R1 camera was usedherein to illustrate the inventive method, although one of ordinaryskill in the art will appreciate the method can be utilized with otherdigital still and/or video cameras. The focusing distance of this cameraranges between the minimal focus distance (e.g., 35 cm for Sony DSC-R1)to infinity.

FIG. 2A depicts a condition 30 in which subject 32 is in focus, whereinthe captured image is the sharpest, as represented by the sharp contrastcurve 34, which is also referred to as the “edge profile” of the stepedge. It will be appreciated that the calibration target, or subject,preferably provides a mechanism for simply determining the sharpness offocus based on contrast. For example in a step-edge target, a clearstep-edge delineation is made between at least two colors, shades,luminances, wherein the sharpness of focus can be readily determinedfrom the sharpness of the contrast profile. It will be appreciated byone of ordinary skill in the art that the target can be configured inany of a number of different ways, in a manner similar to the use ofdifferent chroma keys and color bar patterns in testing differentelements of video capture and output.

FIG. 2B depicts the condition 36 as the image of object 38 becomesincreasingly blurry as the lens moves away from the ‘in-focus’ position,with a resulting sloped contrast curve 40 shown. Generally, when twopictures are taken at two different focal distances, the one takencloser to the subject-distance is sharper than the other. The focaldistances at which the pictures are taken and the amount of the blurdifference between these two pictures can be used to estimate the actualsubject distance, or depth.

Consider a blur difference determination in which two pictures f_(A) andf_(B) are taken at positions A and B, with f_(A) being sharper thanf_(B). The blur change can be modeled by a point spread function P fromposition A to position B according to

f _(A) *P=f _(B)

where * denotes the operation of two dimensional convolution.Furthermore, the point spread function P can be approximated by using aseries of convolutions by a blur kernel K according to

P=K*K* . . . *K.  (1)

For example a blur kernel K may be chosen as

$\begin{matrix}{K = {\frac{1}{64}\begin{pmatrix}1 & 6 & 1 \\6 & 36 & 6 \\1 & 6 & 1\end{pmatrix}}} & (2)\end{matrix}$

in which the amount of blur difference between f_(A) and f_(B) can beevaluated on the basis of how many convolutions are performed in Eq.(1). In actual implementation, the blur difference is more preferablyobtained by an iterative process.

FIG. 3 illustrates an iteration process, herein exemplified with threeiterations performed between picture f_(A) (left) and picture f_(B)(right).

FIG. 4 depicts a matching curve obtained for an image of a step-edgeplaced at a fixed distance (e.g., 100 cm). A first picture of thesequence is taken at the focus distance of infinity, then one picture istaken every time the lens is moved to focus at one depth of fieldcloser, until the focus distance reaches the minimal focus distance.This sequence of pictures is denoted by f₀, f₁, . . . , f_(N-1), where Nis the length of the sequence. In practice, to ensure the sequencecovers the whole focus range, f₀ preferably starts at the distanceslightly further than infinity, and f_(N-1) is slightly closer than thespecified minimal focus distance. These results were achieved using theDSC-R1 camera configured with software for controlling camera steps andsequences.

For a given focal depth, in order to find the relationship between theiteration number and the focus position, a sequence of pictures is takenfor the whole focal range of the camera from which the blur differencebetween every two pictures can be calculated.

It should be appreciated what is meant by the iterations and inparticular negative iteration numbers, as will be seen represented incertain figures (e.g., FIG. 4). Positive iteration numbers indicate thatf_(A) is sharper than f_(B).

2. Correcting Motion Problems with Blur Difference Determinations.

Instead of relying on the use of the norm operator for blur differencecalculations, the present invention applies wavelet transforms after animage convolution stage and compares absolute variances between thesetransforms.

Existing blur matching methods have difficulty with providing accurateresults when motion arises between the images being compared when usingthe blur comparisons based on the norm operator. For example, motion canarise with respect to movement of the subject itself, motion of thecamera, or a combination of these motions. Certain forms of motion, suchas rotation and shape changing, are particularly troublesome forexisting blur matching methods.

The present invention provides an improved mechanism for estimating blurdifferences between images. This comparison between images does notdepend on the pixel correspondence between the two pictures, and is thusreferred to as being motion robust. Use of the norm operator forcalculating differences between two images, generally involves computinga difference between every two pixel values at the correspondinglocations in two images. However, the norm operator approach does notprovide accurate results if motion arises and the object locationschange in the two images.

In general terms evaluating the difference between two images f_(A) andf_(B) can be expressed as a measure G(f_(A),f_(B)). This measureG(f_(A),f_(B)) was previously determined in response to using a normoperator ∥f_(A)−f_(B)∥, such as in certain prior applications of theinventor. However, in the present invention G(f_(A),f_(B)) is determinedin response to performing a wavelet transform on the images and thendetermining the variance between the wavelet transforms. If thevariances S_(A) ² and s_(B) ² are calculated from the waveletcoefficients for f_(A) and f_(B) respectively, then

G(f _(A) ,f _(B))=|s ² _(A) −s _(B) ²|

is the difference measure between two pictures as utilized in thepresent invention for determining blur difference.

Accordingly, the absolute value of the iteration number can becalculated in the present invention by using the equation:

$I_{A\_ B} = {\underset{I}{\arg \mspace{14mu} \min \mspace{14mu} G}\left( {{f_{A}\underset{\underset{I\mspace{14mu} {convulutions}}{}}{\;^{*}K^{*}K^{*}\ldots^{*}K}},f_{B}} \right)}$

and convolutions are performed, expressed by operator * with a kernel K.

The wavelet variance is determined for at least one subband in at leastone level of the wavelet transform structure. In the presentimplementation, the variance was determined for all coefficients at aspecific level, in this case the first level for the image, which by wayof example has a size of 488×273 pixels. This means that the variance iscalculated for all the coefficients in subbands LH₁, HL₁, HH₁ at thisfirst level in this specific implementation.

FIG. 5 depicts a pyramid representation of a wavelet transform. It willbe noted that each level has three subbands: LH, HL, HH. It will beappreciated that each level of wavelet transform generates foursubbands: LL, LH, HL, HH, with the subsequent level wavelet transform istaken on the LL subband of the current level, wherein it transforms thisLL subband into the next level of LL, LH, HL, HH subbands. and so forth.In general, LH, HL, HH subbands provide contrast or sharpnessinformation of the image in different scales of resolutions. The figureillustrates a wavelet transform structure having three levels with LH₁,HL₁, HH₁ in a first level, LH₂, HL₂, HH₂, and LH₃, HL₃, HH₃ in a thirdlevel.

The following first considers each convolution of f_(A) and K, afterwhich a wavelet transform is performed on this blurred image. A varianceis then determined (e.g., calculated) for the wavelet coefficients.According to one implementation the Haar wavelet is utilized forcomputational efficiency and the variance is determined for coefficientsin all subbands (e.g., subbands LH, HL, and HH in FIG. 5) is determinedat a specific wavelet transform level. Implementations of the inventioncan be practiced, however, in response to determining variance for atleast one of the subbands within at least one of the wavelet transformlevels (e.g., levels 1, 2 and/or 3 in the example). It will beappreciated that the inventive technique may be applied in response todifferent types of wavelet computations, consequently implementation isnot limited to the use of the Haar transform.

In addition, the variance may be computed in different ways, such aswith respect to different subband levels, edges and so forth withoutdeparting from the teachings of the present invention. In general, theinvention can be practiced without the difference measure G(f_(A),f_(B))being a norm operator or difference of the variances of the waveletcoefficients. For instance, it can be a difference of contrast orsharpness measures between the two images. However, performance variesin regard to the use of different forms of G(f_(A),f_(B)) computations.One object of the present invention is to provide an accurate andefficient means for determining that blur difference in a motion robustform.

In similar manner, a wavelet transform is applied to f_(B) and againvariance is determined of the wavelet coefficients.

After wavelet variances are determined for each image, then the absolutedifference is determined between the above two variances for the blurredimage of f_(A) and f_(B). This difference is considered the picturedifference and replaces the norm calculation utilized in previous blurdifference calculations. This difference does not depend on the pixelcorrespondence between the two images and therefore provides motionrobust results in response to various type of motions, such as rotationand shape changing.

FIG. 6 illustrates an example embodiment of using convolutions incombination with wavelet transforms and wavelet variances to obtain blurdifferences. It will be appreciated that blur difference determinationis utilized within a number of applications, such as the depthestimation. Blur change is modeled 50 as a point spread function whichare approximated by performing convolutions 52 on the respective images.Wavelet transforms are applied 54 to the images and then a variance isdetermined 56 for each. Difference between the variances is determined58, such as an absolute difference, which is used as a measure of theblur difference. Then the convolutions, wavelet transforms, andvariances are repeated 60, until a desired end condition is met, such asconvergence, reaching a maximum number of convolutions, or other desiredend condition. It should be appreciated that the above convolutions canbe performed in response to a single blur kernel, or in response todecreasing the kernel as convolution progress, as described in anotherinvention by the applicant. In addition, the elements of the presentinvention can be applied to any application which utilizes the normoperator, or similar function, for determining blur difference after aconvolution.

It should be appreciated that as the wavelet transform can be noisesensitive, in at least one embodiment of the invention a low-pass filteris applied to both images f_(A) and f_(B) before blur matching, such asjust before block 52 of FIG. 6. Filtering should be performed just onceon f_(A) and f_(B) before any convolution and wavelet transform.

Determination of blur difference according to this element of theinvention can be applied to a wide range of image processing systems anddevices, such as those related to absolute image focus, comparativeimage focusing, depth estimations, stereoscopic image processing,adjustment of image focus, and other electronic devices which canbenefit from a rapid, accurate and motion robust means of determiningblur difference.

3. Determining the Sharper Image.

It should be noted that when the two pictures are taken, it is unknown apriori which one of f_(A) or f_(B) is sharper, and thus is closer to theproper focal distance. Accordingly the method is configured to computeboth Eqs. 3 and 4 below.

$\begin{matrix}{I_{1} = {\underset{I}{\arg \mspace{14mu} \min}\mspace{14mu} {G\left( {{f_{A}\underset{\underset{I\mspace{14mu} {convulutions}}{}}{\;^{*}K^{*}K^{*}\ldots^{*}K}},f_{B}} \right)}}} & (3) \\{I_{2} = {\underset{I}{\arg \mspace{14mu} \min}\mspace{14mu} {G\left( {{f_{B}\underset{\underset{I\mspace{14mu} {convulutions}}{}}{\;^{*}K^{*}K^{*}\ldots^{*}K}},f_{A}} \right)}}} & (4)\end{matrix}$

If I₁ is larger than I₂, then f_(A) is sharper than f_(B), wherein thevalue of iteration number (as in FIG. 4) will be I_(I). Otherwise if I₂is larger than I₁, then f_(B) is sharper than f_(A), and the value ofiteration number (e.g., as in FIG. 4) will be −I₂. If I₁ and I₂ areequal, then the errors are compared, such as according to the following:

${e_{1} = {G\left( {{f_{A}\underset{\underset{I_{1}\mspace{14mu} {convulutions}}{}}{\;^{*}K^{*}K^{*}\ldots^{*}K}},f_{B}} \right)}};{and}$$e_{2} = {{G\left( {{f_{B}\underset{\underset{I_{2}\mspace{14mu} {convulutions}}{}}{\;^{*}K^{*}K^{*}\ldots^{*}K}},f_{A}} \right)}.}$

If e₁ is smaller than e₂, then f_(A) is sharper than f_(B); otherwise e₂is smaller wherein f_(B) is sharper than f_(A). Alternatively, thevariance values of the wavelet coefficients of f_(A) and f_(B) can besimply used to determine which image is sharper. The one with the largervariance value of the wavelet coefficients is the sharper image, whichcan be performed more readily with similar results than priorapproaches.

The relationship between iteration number and focal positions for thedepth of 100 cm is shown in FIG. 4. The blur difference between everytwo pictures f_(A), and f_(B) for i=0, . . . , N−2 is calculated. The“picture number” axis indicates the image pairs for which the iterationnumber is calculated. For example, picture number 0 means that theiteration number is calculated between f₀ and f₁. It can be seen thatthe absolute value of the number of iterations increases when the lensfocus position moves away from the subject distance. The zero-crossingpoint is where the subject is in focus.

FIG. 7 and FIG. 8 compares the histograms for pictures 138 and 139 fromFIG. 4, wherein significant mismatching is noted. It will be appreciatedthat this mismatching should be removed before subject-distance can beaccurately computed based on blur.

4. Histogram Matching.

To correct for mismatching between images being compared, a matchingprocedure can be performed by modifying one histogram to match the otherone. It will be appreciated that a simple linear matching function canbe utilized, although other functions can be utilized. Pixels aresequentially shifted from one histogram to equalize the number of pixelsof the closest luminance of the other histogram. In response to theshifting of pixels between two histograms, the matching function isdetermined, such as using a least squared error solution. Afterwards thehistogram matching function is applied to the two pictures before thefocusing matching is performed. It should be appreciated that histogrammatching does not work well when the motions become complicated, whereinthis is an optional step of the process which need not be applied toevery application of the instant invention.

FIG. 9 and FIG. 10 depict iteration curves for different subject depthswithout histogram matching being applied before focus matching as shownin the solid lines, and with histogram matching as shown in the dashedlines. FIG. 9 depicts a single example while FIG. 10 depicts theiteration curves for fifteen different distances. The plots for FIG. 10were generated by placing a step-edge at distances of infinity, 1000,500, 300, 200, 150, 125, 100, 80, 70, 60, 50, 45, 40 and 35 cm,respectively. The iteration number I can be written as a function F offocus distance L and subject depth D.

I=F(L,D),  (5)

where L and D are both measured by picture number, which physicallymeans the number of depths of field measured from infinity, or fromwhere picture 0 is defined. Depth estimation is a process to determine Dgiven I and L. On the data shown in FIG. 10, Eq. 5 is used to modeldepth estimation.

The data shown in FIG. 4, FIG. 9 and FIG. 10 manifest significant signalnoise. For instance, in FIG. 4, FIG. 7 and FIG. 8, noticeable outliersare seen at picture number 139. The source of these outliers may includechanges of lighting conditions and aperture variations during thecapturing process, as well as other physical camera and environmentalvariations.

It will be appreciated that in view of the mismatching seen in thesefigures, the histogram matching technique is applied to the imagesbefore the blur difference is calculated. Let h₁ and h₂ denote thehistograms of two different images f₁ and f₂, respectively. Consider h₁as the reference histogram and h₂ as the histogram to be modified tomatch h₁, wherein the following steps are performed.

(1) A pixel mapping matrix w(i,j) is generated.

(2) Setting w(i,j)=0 for every i and j ranging from 0 to the maximumgray level M.

(3) Find the smallest i that satisfies h₁(i)>0, and find the smallest jthat satisfies h₂(j)>0.

(4) If h₂ (j)≧h₁(i), set w(i,j)=h₁(i), update h₂(j) byh₂(j)←h₂(j)−h₁(i), and set h₁(i)=0.

Else If h₂(j)<h₁(i), set w(i,j)=h₂(j), update h₁(i) byh₁(i)←h₁(i)−h₂(j), and set h₂(j)=0.

Steps 3 and 4 are then repeated until both h₁ and h₂ become 0 for allgray levels, which arises in response to the two pictures having thesame number of pixels.

After the mapping matrix w(i,j) is created, a linear matching functionH(x)=ax+b is constructed, such as using a weighted least squaresregression method, where a and b are computed as follows:

$\begin{matrix}{b = \frac{{\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; {{w\left( {i,j} \right)}{\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; \left\lbrack {{w\left( {i,j} \right)}{ij}} \right\rbrack}}}}} - {\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; {\left\lbrack {{w\left( {i,j} \right)}i} \right\rbrack {\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; \left\lbrack {{w\left( {i,j} \right)}j} \right\rbrack}}}}}}{{\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; {{w\left( {i,j} \right)}{\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; \left\lbrack {{w\left( {i,j} \right)}j^{2}} \right\rbrack}}}}} - \left( {\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; {{w\left( {i,j} \right)}j}}} \right)^{2}}} & (6) \\{a = \frac{{\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; \left\lbrack {{w\left( {i,j} \right)}i} \right\rbrack}} - {b{\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; \left\lbrack {{w\left( {i,j} \right)}j} \right\rbrack}}}}{\sum\limits_{i = 0}^{M}\; {\sum\limits_{j = 0}^{M}\; {w\left( {i,j} \right)}}}} & (7)\end{matrix}$

Matrix w(i,j) is generally sparse. In one mode of the method only thenon-zero values and their locations are stored to improve memory andcomputational efficiency.

The histogram matching function H(x) is applied to each pixel of f₂before performing blur matching of the two images. The results of thehistogram matching is shown in FIG. 10.

It should be appreciated that the main purpose of histogram matching isto remove outliers. Even after the matching procedure has been performedit can be seen that the matching curves still exhibit significant noise.Accordingly, after matching is performed the curves are modeledaccording to a polynomial model.

5. Two Dimensional Polynomial Model.

The matching curves described above can be approximated using amulti-dimensional polynomial function, such as a two-dimensional (2-D)polynomial function, to facilitate calculations while removing a largeportion of the mismatching noise seen in FIG. 4 and FIG. 7 through FIG.10.

In this model, the iteration number is a function of lens position andobject distance. The coefficients are determined, for example inresponse to using a least squared error two-dimensional polynomialfitting algorithm. A two-dimensional polynomial is used to model theblur iteration function of Eq. 5.

$\begin{matrix}{I = {\sum\limits_{i = 0}^{m}\; {\sum\limits_{j = 0}^{m}\; {{C\left( {i,j} \right)}L^{i}D^{j}}}}} & (8)\end{matrix}$

The coefficients C(i,j) are determined using a least squaresmultidimensional polynomial fitting method. The degree of thepolynomial, m and n, are chosen depending on the use of specific lensesand applications. Examples of bi-quadratic (m=n=2) and bi-cubic (m=n=3)polynomials are shown in the figures.

By way of a first example, bi-quadratic function coefficients can beused to approximate the fitting algorithm. By way of example and notlimitation, for a bi-quadratic approximation the curves can berepresented by a 3×3 matrix, such as the following.

${C\left( {i,j} \right)} = \begin{matrix}{{- 5.268385}e\text{+}00} & {1.014786e\text{+}01} & {{- 3.073324}e\text{-}02} \\{{- 9.677197}e\text{+}00} & {{- 1.522669}e\text{-}02} & {3.695552e\text{-}04} \\{3.325387e\text{-}02} & {{- 2.438326}e\text{-}04} & {{- 3.833738}e\text{-}07}\end{matrix}$

FIG. 11 is a bi-quadratic fitting curve shown in the dashed lines incomparison with the matching data shown in solid lines. The smooth linesof bi-quadratic curve fitting is in stark contrast to the more jaggedlines for the empirically collected matching data. It will seen that thepolynomial provides a sufficient match with the matching data shown bythe solid lines.

By way of a second example, bi-cubic function coefficients can bealternately utilized to approximate the fitting algorithm. By way ofexample and not limitation, for a bi-cubic approximation the curves canbe represented by a 4×4 matrix, such as the following.

${C\left( {i,j} \right)} = \begin{matrix}{{- 2.096603}e\text{+}01} & {1.414987e\text{+}01} & {{- 1.356138}e\text{-}01} & {5.802068e\text{-}04} \\{{- 1.074841}e\text{+}01} & {{- 1.387527}e\text{-}01} & {4.771262e\text{-}03} & {{- 2.600512}e\text{-}05} \\{8.499311e\text{-}02} & {{- 4.243161}e\text{-}04} & {{- 3.456327}e\text{-}05} & {2.485215e\text{-}07} \\{{- 3.199641}e\text{-}04} & {6.471844e\text{-}06} & {5.348240e\text{-}08} & {{- 6.416592}e\text{-}10}\end{matrix}$

FIG. 12 depicts a bi-cubic fitting curve shown in the dashed lines incomparison with the matching data shown in solid lines. It will be seenthat this bi-cubic polynomial provides a slightly closer match than thatthe bi-quadratic fit shown in FIG. 11.

6. Depth Estimation.

Using the model presented by Eq. 5, the depth estimation method isreadily implemented. First, two images at different focal positions arecaptured, with distance between the focus positions being within onedepth of field. It will be noted that the subject distance is not knownat this moment, as this is what is being estimated. The two picturesused in the process can be captured at any distances as long as thedifference between the focus positions of these two pictures is withinone depth of field. Optionally, noise processing, such as histogrammatching, may be performed on the captured image information prior todetermination of blur difference. The blur difference between thecaptured images as calculated in regards to Eq. 2 through Eq. 5 becomesa single variable polynomial equation. The polynomial equation is solvedfor D, which results in generating an estimated depth of the object,also referred to as subject-distance. It should be noted that D can beconfigured in any desired format, such as an integer or floating pointnumber. For auto focus applications, the lens can be moved to focus atthe estimated distance D, and estimate the new depth in the same manner.The procedure may be repeated until the iteration number converges to 0,or below some desired threshold. It should be appreciated that thisalgorithm may be extended to higher dimensional polynomial models forvarying focal lengths and apertures.

7. General Descriptions of Method and Apparatus.

FIG. 13 illustrates a calibration embodiment, such as would be performedby the manufacturer of a given imaging device, such as a camera. Inblock 70 matching curves are obtained for step-edge images at differentfocal lengths. A two-dimensional model is then created as per block 72to represent the matching curves, by way of example as amulti-dimensional polynomial model. After this calibration process therepresentation of the model, such as its polynomial coefficients, arestored 74, for instance encoded into the non-volatile program memory ofthe camera device.

FIG. 14 illustrates an embodiment of using the multi-dimensionalpolynomial model for depth estimation within a camera device accordingto the present invention. After the calibration process (FIG. 13), themodel is thus available for estimating object depth within the specificcamera device. Represented in block 80, two images are captured (e.g.,pictures taken) at two different focus positions. Histogram matching ispreferably performed on the images as per block 82. Blur difference isthen calculated in block 84 on each image by performing a wavelettransform, determining wavelet variance, and then comparing the absolutedifferences between the variances. After this the polynomial model isused in block 86 to estimate the depth based on the blur difference andthe focus positions at which the two images were captured.

It should be appreciated that a series of depth estimations may beperformed according to the present invention. For example if the methodis utilized in concert with camera focus adjustment, then as the camerafocus is adjusted, additional image input may be collected and thedistance estimation process performed again (or continuously) to provideincreasing accuracy as the camera nears proper focus whensubject-distance estimates match up with the actual subject-distance.

In order to simplify the focus matching model and to smooth theresponse, it is desirable to eliminate errors arising from changes inphysical camera elements (e.g., aperture variation, optical elementpower and temperature variation, mechanical lens setting fluctuations,and the like) and environmental factors (i.e., lighting, motion,temperature, position and so forth). Although the histogram matchingprocess removed some noise source prior to determining blur difference,there is still a measure of noise which can be eliminated, such as wasseen in FIG. 9. Toward removing additional noise, the focus matchingmodel itself determined in response to the calibration process ispreferably cast into a sufficiently smooth mathematical representation(function). This can representation comprising, according to an elementof the present invention a polynomial function of a desired degree(e.g., 2, 3 or 4 degrees) which can be used for depth estimation. Thus,a function (e.g., polynomial function) is selected to substitute for themodel created based on the empirically collected data. It will beappreciated that the substitute function should provide sufficient curvematching (fit) with the empirical data in order that use of the modelwill render sufficiently accurate distance estimations.

For example, given a lens position and an iteration number, atwo-dimensional polynomial equation becomes a single variable equation.Elements of the invention describe examples of the single variableequation as a quadratic or cubic equation, which can be solved in onestep. It should also be appreciated that the algorithm can be extendedto higher dimensional polynomial functions as desired, for example foruse with different focal lengths and apertures.

FIG. 15 illustrates the histogram matching process which is optionallyperformed, as seen in block 82 of FIG. 14, to remove noise prior tocomputing blur difference. It will be noted that in considering imagessubject to complex motions, the use of histogram matching can actuallyintroduce error, wherein its use depends on the application and theexpected motion of the images. As represented in block 90, histogramsare generated for two pictures obtained at different focus positions.Pixels are sequentially shifted from one histogram to equalize thenumber of pixels of the closest luminance of the other histogram as perblock 92. A histogram matching function is determined using a leastsquared error solution as represented in block 94. It should berecognized that a one dimensional linear functions preferably selectedfor this histogram matching function, by virtue of its simplicity. Itshould be recognized that the histogram matching function and the focusmatching function are different and distinct, the latter being a twodimensional polynomial function.

FIG. 16 illustrates an example embodiment of using convolutions incombination with wavelet transforms and wavelet variances to obtain blurdifferences within an image capture system which estimates actualsubject distance in response to blur difference within a polynomial.Images are captured at two subject images 100 and histogram matching, oran alternative process for removing outliers, optionally performed 102.Actual subject distances are estimated 104 and modeling is performed ofblur change as a point spread function approximated 106 by convolutions.After a convolution on a wavelet transform is applied 108 to the images,wavelet variances are determined. After wavelet transforms are appliedto both images f_(A) and f_(B), and wavelet variances determined, thenthe differences are determined 110, such as preferably a measure ofabsolute differences, between these variances as an estimation of thedifference between the pictures and thus the blur difference. Thepreceding convolutions, wavelet transforms, and variance differences arerepeated 112 using one or more convolution kernels until a desired levelof convergence is reached or other desired threshold condition isreached. Depth is estimated 114 based on a polynomial model which hasbeen previously determined for the camera system. In a system providingautomatic focusing (as opposed to a system providing focus indicators),the focus is adjusted 116 on the image capture device (e.g., camerafocus control), and the above steps iteratively performed 118 until thedesired focus accuracy is obtained. It should be appreciated that thepresent invention can be implemented on a variety of devices and systemswhich are configured to perform any of a number of different forms ofimage processing and/or image capture. By way of example and notlimitation the following describes an embodiment within a camera device.

FIG. 17 illustrates an example embodiment 130 of an image capture device(camera) 130 configured for depth estimation according to the invention.A focus/zoom control 134 is shown coupled to imaging optics 132 ascontrolled by a computer (CPU) 136. Computer 136 performs the depthestimation method in response to instructions executed from memory 138and/or auxiliary memory 140. Shown by way of example for a camera device(e.g., video or still) are an image display 142 and touch screen 144,and non-touch interface 146. However, it should be appreciated that theapparatus and method according to the present invention can beimplemented on various image capture devices which are configured withfocus control, focus indicators, or combinations thereof. It should beappreciated that the calibration process (e.g., FIG. 13) which generatesthe model, such as defined by polynomial coefficients, is performed by acomputer controlled test setup. The depth estimation and focusingcontrol elements of the invention are preferably implemented in thecamera device itself as depicted in FIG. 17, or a similar imagingdevice.

It should be appreciated that the blur difference determinations usingconvolutions combined with wavelet variance based comparisons asdepicted in FIG. 6, histogram matching, polynomial modeling of depthestimation, and autofocus steps would all be preferably performed bycomputer processor 136 in combination with memory 138 and/or auxiliarymemory 140. Although the computer processor and memory are describedabove in relation to an image capture device, it will be recognized thatthe computer and its associated programming may be used to perform theblur difference determination within any electronic device whichperforms image processing.

In regards to the use of a zoom control or other means of changing thelens focal length, also referred to as magnification, it should beappreciated that the camera and/or lens system in use will be preferablycharacterized according to the present invention across its applicablezoom range. For example, characterization of the camera and/or lens willbe performed as described for each discrete focal length of lens settingin a camera having discrete lens selections, or at incremental stepsalong the zoom range of a camera having a continuously selectable zoomcontrol. In this way the estimation of distance to a subject can beperformed for single focal length lenses, as described, or for thosehaving multiple ranges whether continuous ranges (e.g., zoom) ordiscontinuous which is more typically referred to as discrete ranges(e.g., normal/macro setting or other selectable range settings). In aprior section the extension of the two-dimensional (2D) polynomial modelto higher dimensions has been described which provides for various focallengths (different zoom positions) and apertures. By way of example andnot limitation, Eq. (5) can be rewritten as I=F(L,D,Z,A) where Z isfocal length, and A is the aperture to provide a four-dimensionalpolynomial model.

The present invention provides methods and apparatus of depth estimationusing blur difference determinations based on wavelet transform andabsolute variance comparisons which are motion robust. The teachingsherein can be applied to a number of systems including cameras and anyform of image capture device, in particular those configured forautomatically detecting and/or adjusting focus. It should also beappreciated that the teachings can be applied without limitation tosystems which process images from a separate camera device, or otherimage source, and in such systems which require blur differencecomparisons, distance determinations, and/or focus control.

As can be seen, therefore, the present invention includes the followinginventive embodiments among others:

1. An image capture apparatus, comprising: (a) an imaging device; (b) acomputer processor coupled to the imaging device; (c) memory coupled tosaid computer processor configured for retaining programming executableon said computer processor; (d) a focus matching model retained in saidmemory; and (e) programming executable on said computer processor forcarrying out the steps comprising, (e)(i) capturing multiple objectimages, (e)(ii) performing convolutions to model blur changes as a pointspread function between said multiple object images, (e)(iii)determining blur difference within each convolution in response toperforming a wavelet transform, obtaining wavelet variance and comparingdifferences of wavelet variance for said multiple object images, and(e)(iv) performing depth estimation in response to said convolutionswithin said focus matching model.

2. The apparatus of embodiment 1, wherein said focus matching model isbased on imaging calibration targets obtained at different focallengths.

3. The apparatus of embodiment 1, wherein programming executable on saidcomputer processor is configured for performing said convolutions by atleast one size of convolution kernel.

4. The apparatus of embodiment 1, wherein programming executable on saidcomputer processor is configured for determining said differences ofwavelet variance in response to a determination of absolute waveletdifferences.

5. The apparatus of embodiment 1, wherein programming executable on saidcomputer processor is configured for determining said wavelet variancein at least one wavelet subband and at least one wavelet transformlevel.

6. The apparatus of embodiment 1, wherein programming executable on saidcomputer processor is configured for determining said wavelet variancein all wavelet subbands in at least one wavelet transform level.

7. The apparatus of embodiment 1, wherein said focus matching modelutilizes a polynomial function to reduce mismatching noise.

8. The apparatus of embodiment 7, wherein coefficients of the polynomialfunction are stored in said memory.

9. The apparatus of embodiment 1, wherein programming executable on saidcomputer processor is further configured for performing histogrammatching of the object images to reduce noise from outliers betweenfocal positions prior to inputting the blur differences into the focusmatching model.

10. The apparatus of embodiment 9, wherein programming executable onsaid computer processor is configured for performing said histogrammatching in response to steps comprising: (a) sequentially shiftingpixels from a first histogram to a second histogram to equalize thepixels of closest luminance; and (b) approximating histogram matchingutilizing a linear matching function; (c) wherein noise effects arereduced which have been introduced into said focus matching model inresponse to undesired physical and environmental variations.

11. The apparatus of embodiment 1, wherein the focus matching model isgenerated by performing a calibration process on said apparatus in whicha series of calibration target images are obtained for registeringproper focus, focus curves are obtained for the series of calibrationtarget images; and a multi-dimensional model generated based on matchingthe focus curves for the series of calibration target images.

12. The apparatus of embodiment 1, wherein the imaging device comprisesa still image camera, a video image camera, or a combination still andvideo image camera.

13. The apparatus of embodiment 1, further comprising: (a) a focuscontrol element coupled to the imaging device; (b) programmingexecutable on said computer processor for adjusting said focus controlelement in response to performing depth estimation on object imagesbased on inputting blur differences detected between object images intosaid focus matching model.

14. An image capture apparatus, comprising: (a) an imaging device; (b) acomputer processor coupled to the imaging device; (c) memory coupled tosaid computer processor configured for retaining programming executableon said computer processor; (d) a focus matching model based on imagingcalibration targets at different focal lengths which is retained in saidmemory, and (e) programming executable on said computer processor forcarrying out the steps of, (e)(i) capturing multiple object images,(e)(ii) performing convolutions by at least one size of convolutionkernel to model blur changes as a point spread function between saidmultiple object images, (e)(iii) determining blur difference within eachconvolution in response to performing wavelet transform, obtainingwavelet variance and comparing differences of wavelet variance in atleast one wavelet subband and at least one wavelet transform level, forsaid multiple object images, and (e)(iv) performing depth estimation inresponse to said convolutions within said focus matching model.

15. The apparatus of embodiment 14, wherein programming executable onsaid computer processor is configured for determining said differencesof wavelet variance in response to a determination of absolute waveletdifferences.

16. The apparatus of embodiment 14, wherein programming executable onsaid computer processor is configured for determining said waveletvariance in at least one wavelet subband and at least one wavelettransform level.

17. The apparatus of embodiment 14, wherein programming executable onsaid computer processor is configured for determining said waveletvariance in all wavelet subbands in at least one wavelet transformlevel.

18. The apparatus of embodiment 14, wherein said focus matching modelutilizes a polynomial function, whose coefficients are stored in saidmemory, to reduce mismatching noise.

19. The apparatus of embodiment 14, wherein programming executable onsaid computer processor is further configured for performing histogrammatching of the object images to reduce noise from outliers betweenfocal positions prior to inputting the blur differences into the focusmatching model.

20. A method of automatic estimation of camera-to-object focal depth,comprising: (a) generating a multi-dimensional focus matching model inresponse to detecting blur differences between multiple images of acalibration subject captured at different focal distances; (b) capturingmultiple object images; (c) determining blur differences between themultiple object images in response to convolutions which model blurchanges as a point spread function between said multiple object images;(d) determining blur difference within each convolution in response toperforming wavelet transform, obtaining wavelet variance and comparingdifferences of wavelet variance for said multiple object images, and (e)performing depth estimation in response to said convolutions within saidfocus matching model.

Embodiments of the present invention may be described with reference toflowchart illustrations of methods and systems according to embodimentsof the invention. These methods and systems can also be implemented ascomputer program products. In this regard, each block or step of aflowchart, and combinations of blocks (and/or steps) in a flowchart, canbe implemented by various means, such as hardware, firmware, and/orsoftware including one or more computer program instructions embodied incomputer-readable program code logic. As will be appreciated, any suchcomputer program instructions may be loaded onto a computer, includingwithout limitation a general purpose computer or special purposecomputer, or other programmable processing apparatus to produce amachine, such that the computer program instructions which execute onthe computer or other programmable processing apparatus create means forimplementing the functions specified in the block(s) of theflowchart(s).

Accordingly, blocks of the flowcharts support combinations of means forperforming the specified functions, combinations of steps for performingthe specified functions, and computer program instructions, such asembodied in computer-readable program code logic means, for performingthe specified functions. It will also be understood that each block ofthe flowchart illustrations, and combinations of blocks in the flowchartillustrations, can be implemented by special purpose hardware-basedcomputer systems which perform the specified functions or steps, orcombinations of special purpose hardware and computer-readable programcode logic means.

Furthermore, these computer program instructions, such as embodied incomputer-readable program code logic, may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable processing apparatus to function in a particular manner,such that the instructions stored in the computer-readable memoryproduce an article of manufacture including instruction means whichimplement the function specified in the block(s) of the flowchart(s).The computer program instructions may also be loaded onto a computer orother programmable processing apparatus to cause a series of operationalsteps to be performed on the computer or other programmable processingapparatus to produce a computer-implemented process such that theinstructions which execute on the computer or other programmableprocessing apparatus provide steps for implementing the functionsspecified in the block(s) of the flowchart(s).

Although the description above contains many details, these should notbe construed as limiting the scope of the invention but as merelyproviding illustrations of some of the presently preferred embodimentsof this invention. Therefore, it will be appreciated that the scope ofthe present invention fully encompasses other embodiments which maybecome obvious to those skilled in the art, and that the scope of thepresent invention is accordingly to be limited by nothing other than theappended claims, in which reference to an element in the singular is notintended to mean “one and only one” unless explicitly so stated, butrather “one or more.” All structural and functional equivalents to theelements of the above-described preferred embodiment that are known tothose of ordinary skill in the art are expressly incorporated herein byreference and are intended to be encompassed by the present claims.Moreover, it is not necessary for a device or method to address each andevery problem sought to be solved by the present invention, for it to beencompassed by the present claims. Furthermore, no element, component,or method step in the present disclosure is intended to be dedicated tothe public regardless of whether the element, component, or method stepis explicitly recited in the claims. No claim element herein is to beconstrued under the provisions of 35 U.S.C. 112, sixth paragraph, unlessthe element is expressly recited using the phrase “means for.”

1. An image capture apparatus, comprising: an imaging device; a computerprocessor coupled to the imaging device; memory coupled to said computerprocessor configured for retaining programming executable on saidcomputer processor; a focus matching model retained in said memory, andprogramming executable on said computer processor for carrying out stepscomprising: (i) capturing multiple object images; (ii) performingconvolutions to model blur changes as a point spread function betweensaid multiple object images; (iii) determining blur difference withineach convolution in response to performing a wavelet transform,obtaining wavelet variance and comparing differences of wavelet variancefor said multiple object images; and (iv) performing depth estimation inresponse to said convolutions within said focus matching model.
 2. Anapparatus as recited in claim 1, wherein said focus matching model isbased on imaging calibration targets obtained at different focallengths.
 3. An apparatus as recited in claim 1, wherein programmingexecutable on said computer processor is configured for performing saidconvolutions by at least one size of convolution kernel.
 4. An apparatusas recited in claim 1, wherein programming executable on said computerprocessor is configured for comparing said differences of waveletvariance in response to an absolute difference of wavelet variance. 5.An apparatus as recited in claim 1, wherein programming executable onsaid computer processor is configured for determining said waveletvariance in at least one wavelet subband and at least one wavelettransform level.
 6. An apparatus as recited in claim 1, whereinprogramming executable on said computer processor is configured fordetermining said wavelet variance in all wavelet subbands in at leastone wavelet transform level.
 7. An apparatus as recited in claim 1,wherein said focus matching model utilizes a polynomial function toreduce mismatching noise.
 8. An apparatus as recited in claim 7, whereincoefficients of the polynomial function are stored in said memory.
 9. Anapparatus as recited in claim 1, wherein programming executable on saidcomputer processor is further configured for performing histogrammatching of the object images to reduce noise from outliers betweenfocal positions prior to inputting the blur differences into the focusmatching model.
 10. An apparatus as recited in claim 9, whereinprogramming executable on said computer processor is configured forperforming said histogram matching in response to steps comprising:sequentially shifting pixels from a first histogram to a secondhistogram to equalize the pixels of closest luminance; and approximatinghistogram matching utilizing a linear matching function; wherein noiseeffects are reduced which have been introduced into said focus matchingmodel in response to undesired physical and environmental variations.11. An apparatus as recited in claim 1, wherein the focus matching modelis generated in response to a calibration process upon said apparatus inwhich a series of calibration target images are obtained for registeringproper focus, with focus curves obtained for the series of calibrationtarget images; and in which a multi-dimensional model is generated basedon matching the focus curves for the series of calibration targetimages.
 12. An apparatus as recited in claim 1, wherein the imagingdevice comprises a still image camera, a video image camera, or acombination still and video image camera.
 13. An apparatus as recited inclaim 1, further comprising: a focus control element coupled to theimaging device; and programming executable on said computer processorfor adjusting said focus control element in response to performing depthestimation on object images based on inputting blur differences detectedbetween object images into said focus matching model.
 14. An imagecapture apparatus, comprising: an imaging device; a computer processorcoupled to the imaging device; memory coupled to said computer processorconfigured for retaining programming executable on said computerprocessor; a focus matching model based on imaging calibration targetsat different focal lengths which is retained in said memory, andprogramming executable on said computer processor for carrying out stepscomprising: (i) capturing multiple object images; (ii) performingconvolutions by at least one size of convolution kernel to model blurchanges as a point spread function between said multiple object images;(iii) determining blur difference within each convolution in response toperforming wavelet transform, obtaining wavelet variance and comparingdifferences of wavelet variance in at least one wavelet subband and atleast one wavelet transform level, for said multiple object images; and(iv) performing depth estimation in response to said convolutions withinsaid focus matching model.
 15. An apparatus as recited in claim 14,wherein programming executable on said computer processor is configuredfor comparing said differences of wavelet variance in response to anabsolute difference of wavelet variance.
 16. An apparatus as recited inclaim 14, wherein programming executable on said computer processor isconfigured for determining said wavelet variance in at least one waveletsubband and at least one wavelet transform level.
 17. An apparatus asrecited in claim 14, wherein programming executable on said computerprocessor is configured for determining said wavelet variance in allwavelet subbands in at least one wavelet transform level.
 18. Anapparatus as recited in claim 14, wherein said focus matching modelutilizes a polynomial function, whose coefficients are stored in saidmemory, to reduce mismatching noise.
 19. An apparatus as recited inclaim 14, wherein programming executable on said computer processor isfurther configured for performing histogram matching of the objectimages to reduce noise from outliers between focal positions prior toinputting the blur differences into the focus matching model.
 20. Amethod of automatic estimation of camera-to-object focal depth,comprising: generating a multi-dimensional focus matching model inresponse to detecting blur differences between multiple images of acalibration subject captured at different focal distances; capturingmultiple object images; determining blur differences between themultiple object images in response to convolutions which model blurchanges as a point spread function between said multiple object images,determining blur difference within each convolution in response toperforming wavelet transform, obtaining wavelet variance and comparingdifferences of wavelet variance for said multiple object images, andperforming depth estimation in response to said convolutions within saidfocus matching model.